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Abstract 

Background: There is a pressing need for greater attention to patient-centered health behavior and psychosocial 
issues in primary care, and for practical tools, study designs and results of clinical and policy relevance. Our goal is 
to design a scientifically rigorous and valid pragmatic trial to test whether primary care practices can systematically 
implement the collection of patient-reported information and provide patients needed advice, goal setting, and 
counseling in response. 

Methods: This manuscript reports on the iterative design of the My Own Health Report (MOHR) study, a cluster 
randomized delayed intervention trial. Nine pairs of diverse primary care practices will be randomized to early or 
delayed intervention four months later. The intervention consists of fielding the MOHR assessment - addresses 10 
domains of health behaviors and psychosocial issues - and subsequent provision of needed counseling and 
support for patients presenting for wellness or chronic care. As a pragmatic participatory trial, stakeholder groups 
including practice partners and patients have been engaged throughout the study design to account for local 
resources and characteristics. Participatory tasks include identifying MOHR assessment content, refining the study 
design, providing input on outcomes measures, and designing the implementation workflow. Study outcomes 
include the intervention reach (percent of patients offered and completing the MOHR assessment), effectiveness 
(patients reporting being asked about topics, setting change goals, and receiving assistance in early versus delayed 
intervention practices), contextual factors influencing outcomes, and intervention costs. 

Discussion: The MOHR study shows how a participatory design can be used to promote the consistent collection 
and use of patient-reported health behavior and psychosocial assessments in a broad range of primary care 
settings. While pragmatic in nature, the study design will allow valid comparisons to answer the posed research 
question, and findings will be broadly generalizable to a range of primary care settings. Per the pragmatic 
explanatory continuum indicator summary (PRECIS) framework, the study design is substantially more pragmatic 
than other published trials. The methods and findings should be of interest to researchers, practitioners, and policy 
makers attempting to make healthcare more patient-centered and relevant. 

Trial registration: Clinicaltrials.gov: NCT01 825746 

Keywords: Patient-reported measures, Health behaviors, Mental health, Pragmatic trial, Electronic health record, 
Primary care 
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Background 

There is significant interest in transforming health infor- 
mation technology to be more patient-centered [1-3]. 
To further integrate these movements toward patient- 
centeredness and information technology-supported 
healthcare, one key element that has been missing is a 
set of brief, practical patient-reported items that are rele- 
vant to and actionable by patients and their healthcare 
team [4,5]. In 2011, 93 national primary care, public 
health, health behavior, and psychosocial experts as well 
as patients engaged in a rigorous three-phase process to 
identify evidence-based, patient-reported measures that 
if routinely collected could be used to improve health 
and monitor health status [6]. The process included 
assembling all possible candidate measures, using a grid- 
enabled measures wild platform to identify key measures 
for consideration [7], and a town hall meeting to finalize 
the measures. From this process, 17 brief, feasible 
screening questions evaluating 10 domains of health 
behaviors and psychosocial problems were identified [8]. 

Before these patient-reported measures are fully inte- 
grated into health risk assessments or routinely inte- 
grated into electronic health records (EHRs) and patient 
portals, research is needed to evaluate relevance, feasibil- 
ity, and impact on care [9-11]. Randomized pragmatic 
implementation studies are one way to efficiently and 
rapidly make this evaluation [12]. Key to pragmatic stud- 
ies is stakeholder involvement, of both clinicians and pa- 
tients, and outcomes important to potential adoption 
settings. Table 1 summarizes the key characteristics of 
pragmatic trials compared to traditional efficacy trials in 
terms of measures, costs, focus and other dimensions. 
As shown, there are major differences in the formulation 
of study questions, methods issues receiving the greatest 



priority, outcomes and analysis, and level of stakeholder 
involvement. Despite the differences, if properly designed, 
both study designs can yield scientifically valid answers to 
posed research questions. The Consolidated Standards of 
Reporting Trials (CONSORT) group has also developed a 
reporting standard and a helpful summary figure, called 
the Pragmatic Explanatory Continuum Indicator Sum- 
mary (PRECIS), to report the extent to which an interven- 
tion study is pragmatic versus explanatory on 10 
dimensions [13,14]. 

The purposes of this manuscript are to describe the 
background, engagement of stakeholders, characteristics 
of participating primary care practices, study design in 
relation to the pragmatic explanatory continuum, key 
outcomes, and automated tool and feedback system de- 
veloped for the My Health Outcomes Report (MOHR) 
study. 

Methods 

Overall design 

The MOHR study is a paired, cluster (practice level) 
randomized, non-blinded, practical, implementation 
study that uses a delayed intervention design with 9 
pairs of primary care practices (18 total). The design 
combines elements of pragmatic trials, implementation 
science, systems science, and mixed methods ap- 
proaches, and engages stakeholders as partners through- 
out the study design, implementation, evaluation and 
subsequent dissemination. One practice in each pair will 
be randomized to early implementation, and the second 
to delayed implementation four months later. Imple- 
mentation will consist of each practice defining a patient 
population and workflow and then fielding the system- 
atic collection of the patient reported measures in daily 



Table 1 Distinguishing differences between pragmatic and traditional clinical efficacy trials 

Pragmatic study Traditional clinical efficacy 



Stakeholder Engaged in all study phases including study design, conducting the study, 
involvement collecting data, interpreting results, disseminating findings 

Research Includes internal and external validity, design fidelity and local adaptation, 
Design real life settings and populations, contextual assessments 

Outcomes Reach, effectiveness, adoption, implementation, comparative effectiveness, 
sustainability 

Measures Brief, valid, actionable with rapid clinical utility, feasible in real world and 
low-resource settings 

Costs Assessments include intervention costs and replication costs in relation 

to outcomes 

Data Source May include existing data (electronic health records, administrative data) 
and brief patient reports. 

Analyses Process and outcome analyses relevant to stakeholders and from different 
perspectives 

Availability of Rapid learning and implementation 
findings 



Limited engagement, often in response to investigator 
ideas or study subjects 

Focus on limiting threats to internal validity, typically 
uses randomized controlled trial, participants and 
settings typically homogenous 

Efficacy, mechanism identification, component analysis 



Validated measures that minimize bias, focus on 
internal consistency and theory rather than clinical 
relevance 

Often not collected or reported 

Data generation and collection part of clinical trial 

Specified a priori and typically restricted to investigator 
hypotheses 

Delay between trial completion and analytic availability 
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care. The primary outcomes will be implementation 
measures and costs that are relevant to potential 
adopting and funding agencies [15-19]. Patient goal- 
setting and clinical follow-up in the 10 health behavior 
and psychosocial domains will be compared between 
early and delayed intervention sites prior to implementa- 
tion of the MOHR assessment in delayed intervention 
sites, thus delayed intervention practices effectively serve 
as control practices. Feasibility will be assessed by moni- 
toring implementation in both early and delayed imple- 
mentation sites (see Figure 1 for project overview and 
timeline). The project is funded through a unique part- 
nership of grant supplementation from the National 
Cancer Institute (NCI), the Agency for Healthcare 
Research and Quality (AHRQ), and the Office of 
Behavioral Social Sciences Research (OBSSR). 

Research teams and study settings 

To carry out this pragmatic trial, partnerships were cre- 
ated between local research teams at eight nationally 



distributed academic institutions that manage a practice- 
based research network (PBRN) or participate in the 
Cancer Prevention and Control Research Network 
(CPCRN) [20,21]. Given the network affiliations, each 
research team has expertise in partnering with their 
practice and patient stakeholders. Accordingly, research 
teams are charged with helping their partner practices 
implement the patient-reported assessment and carry 
out the MOHR study protocol, while balancing the need 
for adaptation to ensure implementation is relevant to 
local culture and practice flow (see 'Balancing fidelity 
and adaption,' below) [22]. To coordinate activities 
across the project, a MOHR planning committee com- 
posed of representatives from all local research teams 
and practices will have joint bimonthly meetings for 
planning (pre-implementation) and to share successes 
and challenges (post-implementation). The organization 
and coordination of activities is depicted in Figure 2. 

Research teams identified one or two pairs of primary 
care practices within their network that were similar 
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Figure 1 MOHR study overview and timeline. The study started in January 2013. The approximate implementation date for the early 
implementation practices is March 2013 and July 2013 for the delayed implementation practices. (IRB = institutional review board, EIS = early 
implementation sites, DIS = delayed implementation sites). 
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Figure 2 MOHR study organization and coordination. The four general partners include funders, project coordination, local research teams, 
and study practices. The funders provide project coordinators general input into the study implementation and design. Overall project 
coordination includes the five working groups, MOHR planning committee, and the coordination center. The local research teams directly 
coordinate with the nine practice pairs to carry out and support the MOHR implementation and evaluation. (ACORN = Virginia Ambulatory Care 
Outcomes Research Network, UV = University of Vermont, UNC = University of North Carolina, UCLA = University of California, Los Angeles, and 
UTH = University of Texas Houston). 



with respect to practice type (e.g., FQHC or PBRN, fam- 
ily practice or internal medicine), practice ownership, 
geographic region, EHR infrastructure, and patient 
population served. Practice pairs were purposefully se- 
lected to represent the diversity of primary care settings 
and populations to ensure greater generalizability of re- 
sults. Practices were randomly allocated to the early or 
delayed condition by coin toss. To conceal allocation, 
the researchers randomizing practices (i.e., project co- 
ordination members) were blinded to practice name; 
practices were labeled by number in paired blocks (by 
local research teams), and identity was revealed to all 
participants one week later during a joint call. 

Table 2 presents the characteristics of the 18 partici- 
pating practices. The PBRN practices tend to have 
higher proportions of insured patients and fewer ethnic 
minorities compared to the FQHCs. Most practices are 
small to medium in size, with one to six providers and 
seeing 1,500 to 10,000 unique adult patients annually. 
All practices, except two, have EHRs. While nearly half 



of practices have initiated or received patient-centered 
medical home designations, most report limited experi- 
ence with behavioral health and varying levels of experi- 
ence with quality improvement. 

Work groups 

To facilitate rapid, collaborative decision making on im- 
portant issues relevant to the MOHR study design and 
implementation, work groups were formed consisting of 
research team members, clinicians, funding agency 
representatives, and content experts. Each work group is 
responsible for effectively solving issues related to their 
specific MOHR study area and bringing proposals to the 
larger collaborative for approval. Work groups include: 
Web Tool and Practice Training Work Group - to de- 
sign and test an electronic version of the comprehen- 
sive behavioral and psychosocial risk assessment, 
www.MyOwnHealthReport.org, for practices to use 
and to create training sessions to prepare practices to 
address topics identified by the MOHR assessment; 
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Table 2 Participating practice characteristics 

Site State Setting Patients seen per year Provider FTEs Rooming staff FTE Patient ethnicity and race Insurance type 















Latino 


Black 


Medicare 


Medicaid 


None 


1 


VA 


S 


5,000 


2 


4 


15% 


10% 


5% 


0% 


30% 


2 


VA 


S 


1,500 


1 


2 


20% 


10% 


1% 


0% 


9% 


3 


VA 


R 


2,500 


1.6 


7 


1% 


49% 


12% 


1% 


49% 


4 


VA 


S 


5,200 


4 


11 


2% 


18% 


15% 


2% 


18% 


5 


VA 


U 


3,700 


5.9 


17.3 


2% 


39% 


24% 


2% 


39% 


6 


VA 


U 


3,400 


5.3 


16.9 


1% 


17% 


26% 


1% 


17% 


7 


CA 


R 


3,500 


5.5 


15 


3% 


1% 


13% 


3% 


1% 


8 


CA 


R 


5,400 


7 


25.5 


13% 


2% 


12% 


13% 


2% 


9 


VT 


R 


9,500 


5 


13.5 


1% 


5% 


13% 


1% 


5% 


10 


VT 


R 


10,000 


5 


14 


1% 


2% 


15% 


1% 


2% 


11 


NC 


R 


1,100 


4.5 


12 


2% 


60% 


49% 


2% 


60% 


12 


NC 


R 


7,500 


3.5 


10 


40% 


60% 


10% 


10% 


70% 


13 


CA 


U 


2,040 


1 


7 


75% 


25% 


5% 


45% 


50% 


14 


CA 


U 


2,180 


2 


6 


75% 


25% 


5% 


45% 


50% 


15 


TX 


R 


4,800 


2 


6 


48% 


23% 


2% 


48% 


23% 


16 


TX 


R 


3,800 


2 


6 


23% 


32% 


2% 


23% 


32% 


17 


TX 


U 


2,800 


3 


19 


82% 


6% 


1% 


82% 


6% 


18 


TX 


U 


2,800 


3.6 


12 


80% 


5% 


1% 


80% 


5% 


Notes: 



Practices 1-10 belong to a practice based research network and practices 11-18 belong to the Cancer Prevention and Control Research Network. 

All practices except Site 2 and 3 have an electronic health record. 

All practices have experience with prior research or quality improvement projects. 

FTE Full Time Equivalent. 

5 Suburban, R Rural, U Urban. 



Patient Experience Survey Work Group - to design the 
Patient Experience Survey (see 'Outcomes assessment', 
below) for assessing main patient-level outcomes; 
Clinical Context Work Group - to identify and evalu- 
ate how key practice, patient, and environmental con- 
textual issues impact implementation; Cost Collection 
and Analyses Work Group - to measure the practice 
costs of implementing the MOHR assessment; and Publi- 
cation, Presentation, and Relations Work Group - to 
promote opportunities for participants to present and 
publish findings, coordinate with relevant stakeholder 
entities, and create a repository of materials, tools, and 
disseminated findings. 

Intervention 

The intervention being studied is the systematic imple- 
mentation and fielding of the MOHR health behavior 
and psychosocial assessment, including the delivery of 
brief customized patient and provider feedback reports. 
Each local research team worked with their practice pair 
champions and key stakeholders to determine the most 
effective workflow for administering the MOHR assess- 
ment, including: (a) which patients would be invited to 
complete the assessment, (b) when the assessment 



would be completed (e.g., two weeks before a visit, im- 
mediately before a visit), (c) where the assessment would 
be completed (home or practice), (d) whether the assess- 
ment would be electronic or paper based, and (e) how 
clinicians would receive MOHR feedback summaries 
within the context of office visits. This is consistent with 
the original NIH consensus panels goal of identifying 
and standardizing patient reported elements to be incor- 
porated into EHRs while minimizing staff burden and 
enhancing consistency of implementation. As a result, 
the practice pairs were encouraged to field the electronic 
version of the MOHR assessment if possible [8]. 

Because it was not feasible to fully integrate the 
MOHR assessment into each of the diverse practices 
EHR and patient portal, a stand alone, publically avail- 
able website (www.MyOwnHealthReport.org) was cre- 
ated. The MOHR website is partially integrated with the 
study practices' existing EHRs and administers the 17 
health behavior and psychosocial screening questions 
(including basic demographics) as defined by the NIH 
consensus panel [6,8]. In response to positive depression, 
anxiety, alcohol, and drug screening questions, the 
website seamlessly prompts the patient to complete 
more in-depth assessments, including the Patient Health 
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Questionnaire (PHQ9) [23], the Generalized Anxiety 
Disorder (GAD) questionnaire [24,25], the Alcohol Use 
Disorders Identification Test (AUDIT-C) [26], and the 
Drug Abuse Screening Test (D AST- 10) [27], respect- 
ively. The MOHR website scores the patients responses 
and characterizes them as being of no concern' , some 
concern', or 'high concern' based on norms and national 
guidelines. For measures with some' or 'high concern', pa- 
tients are asked to select which topics they are ready to 
change and/or discuss with their doctor and select the one 
topic they feel is most important to address [28-31]. 

After completing this process, patients are taken to a 
feedback page to review, download and print (see 
Figure 3). Health domains in which they are doing well 
are highlighted and reinforced, initial recommendations 
for changes and improvements are given for domains 
where improvements are needed, and space is provided 
to list any questions, decide on and create up to three 
SMART goals (specific, measurable, achievable, realistic 
and timely), and outline follow-up steps to help achieve 
their goals [32,33]. When patients have finished 
reviewing their feedback page and leave the website, a 
clinician summary is also sent to the patient's primary 
care team. 

Practice implementation preparation 

Practices are being prepared to implement the MOHR 
assessment through a series of calls and meetings with 
the site champions, an educational webinar with an 
interactive call in' and case scenarios, support materials, 
and on-site consultation for follow-up. The webinars 
have been designed using the principles of adult learn- 
ing, including multiple modalities, reinforcement, mod- 
eling and observation. The follow-up consultations have 
been modeled on academic detailing that entails a brief 
face-to-face intervention with the clinician, repeated at 
periodic intervals [34-36]. Academic detailing has been 
found effective in improving other aspects of clinical 
care such as cancer screening and chronic disease ma- 
nagement [37-43]. 

The webinars present the evidence supporting the 
MOHR assessment, strategies for successful practice im- 
plementation of risk assessments, and approaches to 
counseling and supporting patients as they address iden- 
tified problem behaviors and psychosocial issues. The 
webinars, as academic detailing, distill the evidence into 
brief, practical messages that are relevant to and action- 
able for primary care clinicians. The webinar and local 
research team consultations seek to change staff atti- 
tudes and beliefs toward implementing the MOHR as- 
sessment though persuasive communications, including 
case examples, tailored feedback and reinforcement [44]. 
Feedback and reinforcement will include weekly practice 
reports on the number of patients completing the 



MOHR assessment and bimonthly meetings between 
practice champions and local research teams to share 
cross project implementation experiences and review the 
local implementation process. 

Data sources 

Outcomes will be collected from three sources: practice 
appointment records, the Patient Experience Survey, and 
patient responses to the MOHR assessment. Appoint- 
ment records will be used for calculating Reach (see 
'Primary outcomes', below). Reach measures are col- 
lected when practices enter the intervention phase, 
months two to six for early intervention practices and 
months seven to nine for delayed intervention practices. 
Appointment records are queried by the practices (elec- 
tronically or manually) for patients who meet inclusion 
criteria, are documented, de-identified, and transferred 
to a central data coordination center every two weeks. 
The Patient Experiences Survey will be used to compare 
the early versus delayed intervention practices on patient 
reports of patient-provider discussions, recommenda- 
tions for referrals, and collaborative goal-setting across 
the 10 health domains. The patient responses to the 
MOHR assessment will be used for calculating patient 
health behavior and psychosocial changes over time. 
This will be assessed by repeated administration of the 
MOHR assessment for the cohort of patients who 
complete the assessment on the website and provide 
email addresses (see 'Outcomes assessment', below). 

The content of the Patient Experience Survey and fiel- 
ding protocol were developed by the working group, 
building on earlier pilot phase experiences and practice 
input, with the intention of balancing scientific rigor and 
patient needs, practice capabilities, and resource avail- 
ability across the diverse settings. The survey questions 
were standardized. The 5As framework (Assess, Advise, 
Agree, Assist and Arrange) served as a guide for devel- 
oping questions for each health domain [45]. Questions 
from the Consumer Assessment of Healthcare Providers 
and Systems (CAHPS) assessing patient trust in their 
healthcare team and perceived provider communication 
style were also included [46]. Practices will mail the Pa- 
tient Experience Survey to patients that the early and de- 
layed intervention practices defined as eligible to receive 
the MOHR assessment two weeks after the patient's of- 
fice visit. To enhance survey response rates, surveys will 
be mailed using a modified Dillman technique in the 
practice's envelope with a cover letter from the patient's 
provider [47,48] . Study information will be included with 
the survey, and completion of the survey will be deemed 
as consent for participating in the survey - our Institu- 
tional Review Board approved a waiver of written 
consent. 
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PatientHeaKhS^Repor, 




YOUR Health Behaviors and Mental Health 





Recommended Score 


Your Score 


Level of Concern 


Ready to Change? 


Want to Discuss? 


Overall Health Rating 

Reason 1 am working too hard at my job 


Good to Excellent 




A Lot 






Body Mass Index 


20-25 


27.7 


Some 






Health Behaviors 


Fruit/Vegetable Intake 


5+/day 


Less than 2/day 


A Lot 






Fast Food Intake 


Less than 1 time/week 


1-3 times/week 


Some 






Soda/Sugary Beverage Intake 


Less than 1/day 


1 to 2 day 


Some 






Physical Activity Participation 


150+ minutes/week 


175 minutes/week 


None 






Sleep 


Never/rarely sleepy 


Often sleepy 


Some 






Alcohol Intake 


Never 




None 






Tobacco use 


No 


Yes 


A Lot 






Illegal Drug/Prescnption Use 


Never misuse 


Never misused 


None 






Mental Health 


Stress 


Less than 5 


8 


A Lot 






AnxietyM'orry 


Not at all/rarely Not at all rarely 


None 






Depression 


Not at all/rarely 


Not at all/rarely 


None 







Most important to you 



Keep up the GOOD Work! 

■ You are meeting or exceeding the physical activity 
recommendations for health. 

■ You said there are few days you feel nervous, anxious, on edge 
or unable to stop or control worrying 

■ You said there are few days you feel down, depressed, hopeless 
or have little interest or pleasure. 

■ You never dnnk too much alcohol 

■ You do not use illegal drugs or prescription medications for non- 
prescrll>ed reasons 



Recommendations to Improve Your Health 



Medium Priority 

■ Excess weight can lead to a number of 
health problems Increase physical 
activity and/or limit the unhealthy food you 
eat to reduce your weight. 

■ Decrease your fast food neals or snacks 
to less than one per week. 

■ Decrease the number of soda or sugary 
drinks you drink to less than 1 per day. 

■ Try to get 7-8 hours of sleep each night 



High Priority 

■ Increase fruits and vegetables to 5 or 
more servings per day 

■ You reported feeling stressed often 
Discuss ways to reduce your stress 

■ Discuss options for decreasing or quitting 
tobacco use 



Notes/Things to Discuss During My Appointment 



Health Goals 



The best goals to set are those that are specific, measurable, achievable, realistic, and timely a~d focus on the who, what, where, when and how you will achieve them in 
order to be able to measure your success List 1-3 goals you have to try to improve your health based on your health update. 

Example Goal: 

What will you do? Decrease fast food by eating out 2-3 less times per week 

How will you do it? Pack a lunch to bring to work 2 times per week and cook dinner one more time a week. 

fly when? Gradually work up to this over the next 3 weeks by decreasing fast food meals by one per week until I reach 3 

Goal #1: 

What will you do? 



How will you do it? 



3, :.her,~ 



Goal »2: 

What will you do? 
How will you do it? 



Goal »3: 

What will you do? 

■, , o\: -'■-> : ' 
fly when? 




Figure 3 Patient summary and feedback from www.MyOwnHealthReport.org. The first page demonstrates the patient's health behavior and 
psychosocial scores, level of concern, and whether the patient reported readiness to change and interest in talking with their doctor. The second 
page includes patient workspace for notes, creation of SMART health goals, and a follow-up plan. 



Krist et al. Implementation Science 2013, 8:73 
http://www.implementationscience.eom/content/8/1/73 



Page 8 of 13 



Outcomes assessment 

The primary outcomes for this study are implementation 
measures [15]. The measures include the practices' im- 
plementation Reach and Effectiveness [49]. Reach mea- 
sures include the percent and characteristics of patients 
that: are defined as eligible to be invited to complete the 
MOHR assessment; are invited to complete the MOHR 
assessment; and complete the MOHR assessment. The 
primary Effectiveness measures are a comparison of 
early versus delayed intervention practice patient report 
on whether they: were asked about the health behavior 
and psychosocial domains; set a goal or action plan with 
their clinician; and arranged any follow-up contact re- 
garding their plan. As early intervention practices will 
have implemented the MOHR assessment and delayed 
intervention practices will not yet have implemented the 
MOHR assessment, they will respectively serve as inter- 
vention and control conditions. The impact on health 
behaviors and psychosocial problems will be assessed by 
repeated administration of the MOHR assessment for 
the subset of patients providing an email address. Patient 
responses will be compared at baseline and four months 
after baseline. Multivariate regressions will assess for 
influences of practice implementation characteristics 
(e.g., electronic versus paper) and patient demographics 
on each outcome. The overall CONSORT diagram 



depicting study participation and data capture are 
depicted in Figure 4. 

Power and sample size analysis 

To determine power, we focused on the patient report as 
to whether any practice personnel worked with her/him 
to set specific goals for change for the 10 behavior and 
mental health topics. For the base power analysis, we as- 
sumed 9 matched pairs of sites, 300 patients mailed the 
Patient Experience Survey from each site, and a 50% re- 
sponse rate (2,700 total respondents). Imputation ana- 
lyses will be conducted to evaluate the impact of missing 
data, but here we assume the loss of up to 50% of the 
sample for power calculation purposes. For the first 
power analysis, we focus on the binary event that at least 
one goal was established (as opposed to none). Assum- 
ing a delayed intervention response rate of 0.3, there is 
90% power to detect a 17.5% improvement in the early 
versus delayed intervention group, and 80% power to de- 
tect a 15% improvement. 

For the second power analyses, we focus on the con- 
tinuous outcome of the number of goals established 
using a hierarchical analysis of covariance. We adjust for 
any baseline differences in variables related to the out- 
come as well as clustering effects of patients within 
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clinic. Assuming an intraclass correlation coefficient 
(ICC) of 0.03, there is 90% power to detect a between 
conditions effect size of 0.47 with 9 sites per treatment 
and a 50% patient response rate, and 80% power to 
detect an effect size of 0.40. Assuming a larger ICC of 
0.05, we have 80% power to detect an effect size of 0.51 
and 90% power to detect a difference of 0.60, given 
the assumptions above. All power analyses were 
performed using the 'Optimal Design Software' pack- 
age (version 3.0). 

Contextual assessments 

An important aspect of a pragmatic trial is that it is 
implemented in real-world' contexts, making it essential 
that these contexts be described to understand differen- 
tial outcomes [50]. Accordingly, local research teams will 
collect data on multiple levels of contextual factors 
[51-53]. Levels will include national/state/local, commu- 
nity, healthcare system, practice and patient [54]. Con- 
textual factors will include practice culture and staffing, 
patient panel characteristics, community characteristics 
and resources, and organizational features [55]. Local re- 
search teams will collect relevant data through struc- 
tured interviews with clinicians and through field 
observations of clinical, social and organizational pro- 
cesses and interactions as practices field the MOHR 
assessment at the baseline, middle and end of the inter- 
vention to assess changes in context. Quantifiable char- 
acteristics will be incorporated from initial practice 
descriptions made during recruitment. Teams will use a 
standardized template to systematically elicit and record 
contextual data. 

Cost assessment 

The MOHR cost assessment will evaluate both the im- 
plementation costs and the replication costs using cost 
assessment procedures that are minimally intrusive and 
burdensome [19]. Using standardized a procedure, local 
research teams will collect each practice's cost estimate 
following brief training. This person will use a combin- 
ation of procedures, including observation of the inter- 
vention and brief standardized structured questions of 
key practice implementation staff. Cost data will be col- 
lected at two time points - early in the implementation 
and once a steady state has been reached. Calculations 
will be made using established expenditure methods, 
and replication costs will be categorized based upon 
different practice and implementation characteristics 
[56-59]. The cost assessments will be conducted from 
the practice perspective and will include costs of recruit- 
ment, promotion, training, supervision, implementation 
and follow-up. Data collected during the trial will be 
used to estimate the cost of a practice replicating the 
MOHR assessment setting under different conditions 



and assumptions. Research costs, downstream costs of 
increased or reduced healthcare utilization, and costs 
from the patient perspective will not be considered. 

Balancing fidelity and adaption 

To maintain internal validity, certain components of the 
intervention were standardized. All participating prac- 
tices will implement these components, and fidelity to 
implementation will be assessed. However, to reflect the 
pragmatic nature of the trial, practices were encouraged 
to adapt aspects of the intervention to their workflow 
and setting [22]. This is necessary because the MOHR 
study purposively includes practices that vary widely 
with respect to patients (mostly minority versus non- 
minority), location (rural versus urban), and EHR and 
patient portal use. Therefore, to ensure successful 
implementations as well as high assessment and survey 
response rates, each practice pair considered its patients' 
needs. The elements that needed to be standardized 
were decided by the overall project steering committee 
and working groups, while decisions about local tailoring 
were made by the practice pairs with support and direc- 
tion from the local research team. 

Standard study elements include the MOHR health 
behavior and psychosocial assessment questions, provi- 
sion of patient and provider feedback summaries based 
on the MOHR assessment, goal-setting materials and 
brief training to support clinicians with counseling, and 
Patient Experience Survey questions. Practice pairs are 
collaboratively deciding which patients to target for the 
MOHR assessment, while attempting to include the lar- 
gest percentage of adult patients that is feasible; timing 
of the MOHR assessment (one to two weeks prior to a 
visit at the patient's home or immediately prior to a visit 
at the practice); mode of the MOHR assessment (website 
or paper), delivery of clinician summary (integrated into 
EHR, faxed, printed); and mode of administering the Pa- 
tient Experience Survey (postal, web-based, telephone or 
a combination). 

Trial status 

The MOHR trial began January 2013 and is expected to 
complete data collection in November 2013. The trial is 
proceeding according to the timeline outlined in Figure 1. 
Currently, the early implementation clinics are in the 
early stages of fielding the MOHR assessment to their 
patients. The delayed intervention clinics will begin im- 
plementation between July and September 2013. 

Discussion 

This manuscript describes the need and plans for an im- 
portant ongoing pragmatic study of the administration 
and use of patient-reported items in primary care, illus- 
trating key features of implementation design with both 
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standardized and flexible aspects reflecting the dual 
realities of research and practice. A central tenant 
throughout the process has been ensuring the relevance 
of the intervention, measures, and methodologies to 
stakeholders [10,12]. 

Figure 5 summarizes the characteristics of the MOHR 
study on the 10 CONSORT PRECIS dimensions. 
Overall, the MOHR study is highly pragmatic, receiving 
completely pragmatic scores on three dimensions (inter- 
vention flexibility, control flexibility, and practitioner 
expertise) and the next most pragmatic rating on the 
seven remaining dimensions. These scores are substan- 
tially more pragmatic than other published trials [60] . 

Implementation science involves controlled research, 
in this case a cluster randomized delayed intervention 
trial, but that is pragmatic and thus more directly rele- 
vant to real-world settings [9,61,62]. By engaging all of 
the stakeholders on the front-end about the importance 
of healthcare process and outcome metrics for 
documenting healthcare quality, the results should be 
more directly usable by both primary care and potential 
funding and reimbursement agencies. This is especially 
so when including practical measures that can be em- 
bedded into clinical systems [63], applying a feasible and 
flexible intervention that provides direct feedback based 
on these practical measures, and assessing outcomes of 
importance to patient, provider, and policy maker stake- 
holders, including costs. 

This type of science is needed because 'evidence alone 
is not enough' [64,65]. In addition to the best evidence, 



as in this case recommendations for primary care-based 
health behavior and psychosocial counseling, other ele- 
ments are needed [66,67]. As noted in the Evidence 
Integration Triangle framework, other necessary compo- 
nents for successful implementation are practical 
measures of progress that can be tracked longitudinally 
(in MOHR, these are the 17 items), and partnership en- 
gagement processes, as illustrated above [64]. 

A major advantage of the MOHR trial for advancing 
both scientific and practice frontiers is its diversity of 
settings. Participating primary care settings include prac- 
tices that already have adopted state-of-the art e-health 
technologies for tracking patients' care, enabling an 
examination of the merger of patient-centered assess- 
ment and intervention approaches and EHRs in high 
resourced settings [68,69]. On the other end of the con- 
tinuum, inclusion of a number of community health 
centers, typically with less access to state-of-the art tech- 
nologies, has generated problem-solving about the best 
strategies for conducting assessment and interventions 
in contexts where staff are challenged to see large 
caseloads of more diverse, low income/low educated 
populations with characteristically complex medical and 
psychosocial needs, and lower levels of health literacy 
and numeracy [70]. The mantra remains consistent: in- 
volving relevant stakeholders is important for both the 
initial adoption as well as the long-term sustainability of 
evidence-based patient assessment and intervention. 
This requires an understanding of the realities of the 
practice flow, an appreciation for the context of patient 
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lives, and how to design practical assessments and feed- 
back mechanisms that can be implemented with min- 
imal resources. 

From a traditional perspective, there are several limita- 
tions or weaknesses to the MOHR design. Scientifically, 
there are several questions the trial will be unable to ad- 
dress. The MOHR assessment will not be linked at the 
individual level to the follow-up Patient Experiences 
Survey, and the design selected precludes the examin- 
ation of the long-term behavioral or biological health 
impact of our intervention. In contrast to more trad- 
itional research studies, the project will not employ ex- 
pert, highly trained staff in leading academic medical 
centers to deliver the intervention; rather the interven- 
tion will be implemented within real-world settings with 
tailoring of strategies to meet the needs of each site. 
Similarly, a decision was made to be as inclusive as pos- 
sible in the patient intake rather than using strict eligi- 
bility criteria selecting for non-complicated, highly 
motivated patients who often fare better in health pro- 
motion programs. Finally, practice participation is not 
limited to practices with excellent state of the art EHRs 
serving technology savvy patients; thus, a difficult bal- 
ance has been sought between maximizing the use of 
technology whenever possible and developing protocols 
applicable to a range of settings. All of these decisions 
could reduce the initial effect size, or produce different 
results across different contexts, subgroups or settings. 
However, despite this possibility, we suggest that the 
project remains a worthwhile investment that will pro- 
vide invaluable information about what works under 
what conditions for what groups for what outcomes'— 
in a relatively short time frame [71]. 

The study design has many strengths, including the 
highly pragmatic nature of the protocol, as evidenced by 
the PRECIS ratings. Given the stakeholder involvement 
throughout, the results should be highly relevant and 
sensitive to both the primary care context as well as to 
the growing emphasis on patient-centered care for im- 
proving population health and well being. Anticipating 
the relevance of the results to diverse primary care prac- 
titioners as well as different healthcare settings and net- 
works, the rapid and iterative processes using mixed 
methods should provide in-depth feedback to more 
quickly inform practice and policy. Inclusion of data on 
reach, implementation under different conditions, as 
well as implementation and replication cost estimates 
should provide the types of transparent information 
needed by potential adopting settings. The tools and ma- 
terials used in the MOHR project, including the stan- 
dardized brief assessment items, the MOHR automated 
assessment and feedback tool, and the assessment proce- 
dures are all in the public domain and available for 
others to use or adapt (www.MyOwnHealthReport.org). 



Replications and extensions are encouraged, and espe- 
cially studies having the scope and funding to evaluate 
issues such as longer-term sustainability, health behavior 
changes, and health outcomes. 

The MOHR study serves as an example of a prag- 
matic, participatory design that leverages resources and 
tailors design elements to a range of local settings, while 
maintaining the essential elements and scientific rigor of 
a randomized comparison trial. This design is necessary, 
given that the MOHR study aims to assess whether it is 
feasible for a broad range of primary care practices to 
systematically patient reported health behavior and psy- 
chosocial assessments. Outcomes will include all neces- 
sary implementation elements including feasibility, 
impact on care, the influence of setting and context, and 
the expense of implementation. These findings should 
be of interest to researchers, practitioners, and policy 
makers attempting to make healthcare more patient- 
centered and relevant. 
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